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The evaluation of the PLATO and TICCIT programs is being designed 
to focus on three major components of the demonstration projects - 
their costs, performance, and educational effectiveness* 

While the thrust of this paper will be to describe the frame- 
work for the evaluation of educational effectiveness, brief discussion 
of the objectives of the cost and performance components is appropriate 
to provide a complete perspective. 
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The cost component of the evaluation will attempt to measure the 
costs and assess the economic effects of each of the systems on the 
0^ participating educational institutions. In particular, attention 

will be given, to identifying instructional costs and cost effective- 
ness of courses presented under CAI conditions, to answering questions 
about the capital, development, and operating costs of each of the 
two systems, and to estimating indirect costs to participating 
institutions resulting from adoption of the pLATO or TICCIT system. 

The technical component of the evaluation will monitor the per- 
formance of each of the two systems so as to discover and explain 
signif ican t— s-t T-eng-t hs , weakn e s s es.,~and—impli.cat-io^s—f or— f utur e -<use-» 
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Particular at^tention will be given to assessment of the system specif- 
ications to ensure adequacy of design; analysis of the stability cf 
operation and the adequacy of documentations, operating procedures, 
and maintainability of the system; and appraisal of the appropriateness 
of the computer system design relative to the state of the art. 
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The educationit 1 component will focus on evaluation of the effect 
of the CAI upon student behavior and performance; assessment of the 
impact of the CAI upon instructors, administrators, and the institu*- 
tion itself; and appraisal of the effect and potential of the methods 
used to produce, operate, and maintain the course materials. 

Educational Analysis 

Despite substantial prior research in computer-assisted instruc- 
tion, inst rractional systems typically lack detailed information 
regarding their impact upon the educational community. The development 
of delivery systems and course materials has, in most cases, proceeded 
without adequate attention to their educational effectiveness. However 
the PLATO and TICCIT demonstrations offer an opportunity to assess 
effects and reaccions stemming from computer-based instructional 
technology and to identify significant educational strengths; and 
weaknesses. The scope of these demonstrations will make possible the 
collection of detailed information which reflects not only cost and 
technical sophistication, but also the effects on achievement and 
educational acceptance. Thus the KSF CAI project extends beyond a 
development exercise to a study of ins true t ional * technology ' s impact 
upon the educational institution, upon students, t eacher and adminis- . 

^^trat_o.rs-« The. _educational-_compo.nen-t.__o-f__the_-P_LAT.O_ahd-_-r^ 

will focus upon the' consumers of educational Innovations in order to 
determine the practical benefits and problems accompanying computer'^ 
based education. • . • 

The following framework for the educat ibnal analysis presents an 
overview of educational effectiveness as related to the PLATO and 
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TICCIT projects. Since "effectiveness" is subject to diverse inter- 
pretations, that concept requires as much prior elaboration and 
specification as possible. The entire context of the evaluation of 
educational effectiveness will be clarified by the delineation of 
four essential issues. For the sake of brevity those issues may be 
posed as four questions that must be answered: the what , who , how , 
and when of the evaluation. Together these basic. questions form the 
dimensions of an analytical approach to educational effectiveness. 

The first task for an educational analysis is to establish what 
areas of* inquiry are appropriate to an evaluation of computer-based 
education. The cognitive and affective domains represent a gross 
classification of this "what" dimension. For the cognitive domain, 
effects upon students^ achievement and behavior are our primary 
interests. For the affective domain, consumers^ reactions to the use 
of an innovative technology will provide important attitudinal infor- 
mation related to educational impact. Through successive refinements 
of both domains in a hierarchical manner, greater precision and detail 
will yield indications of what subtopics and issues are relevant to 
effectiveness. The breadth of coverage Incorporated into such a 
hierarchical specification of the "what" dimension ensures our 
objectivity; detail permits a telling probe of the shibboleths of 
computer-based education. 

The several audiences concerned with the demonstrations constitute 
the "who" dimension of the educational analysis. Because of their 
differing perspectives and preconceptions, each audience serves as 
an important' source of data. Depending on the particular area of 
Inquiry, sources of critical information are the educational instltu- 
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tlon, government^ Industry, and the public. This attention to 
different societal sectors leads not only to the representation and 
documentation of diverse viewpoints on educational effectiveness, 
but also to supportive Information for the cost and technical analyses. 
The expression of affective and cognitive Impact across vested Inter- 
ests and characteristics Is of course fundamental to the concept of 
evaluation. Further elaboration of the "who" dimension permits us 
to establish necessary comparison groups and to represent various 
groups of persons. 

The systematic administration of Instruments in the areas of 
Inquiry and to appropriate audiences also requires a plan of how 
measurement is to be conducted. How Instruments are to be administered 
relates to the available modes for data acquisition. The range of 
available and proposed means for collecting data includes: tests, 
records, questionnaires, online systems, interviews, and observations. 
Each mode has particular demands for instrumentation, ranging from 
classical test attributes (reliability and validity) to the allocation 
of personnel and financial resources. Precisely when an instrument 
should be administered is, similarly, an Important consideration, 
since baseline and concurrent contr 1 procedures .depend upon coordln-* 
atlng measurement with the timing of administrations. Data collection 
at-speclfled time intervals is also critical to obtaining trend 
information a^bout effectiveness over an* extended period of time. 

Certainly there are interrelationships ' among these dimensions 
of the educational analysis, as Indicated in Figure 1. Although 
many combinations of components along these dimensions would be 
inappropriate for our purposes, the individual cells in this initial 
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schematic representation of what > who and how do begin to identify 
Information critical for assessing effectiveness* The accumulation 
of that information will depend, however, upon our first identifying 
1) the area of inquiry; 2) the audience to respond; 3) the instrument 
form used for data acquisition; and 4) the time of administration 
of the instrument • Through such specification, data can be gathered 
that attest to the educational stengths and weaknesses of the demon- 
strations in terms of effects upon usersa 
Dimensions 

A> Areas of Inquiry , To capture the impact of the demonstrations 
on cognitive skills and attitudes, the evaluation must isolate indi- 
cators from the cognitive and affective domains. For the cognitive 
domain indications of instructional effectiveness are to be gathered 
through various achievement measures, including standardized tests, 
objective-based (personalized) tests, and measui^es of course performance.. 
We shall also supplement information on student achievement with bchav** 
ioral data on lesson completion, instructional sequence, and other 
descriptions of cognitive approach toward the subject matter. Activi- 
ties of students and instructors will serve as indicators of ancillary 
effects beyond instructional outcomes, such as changes in the distri- 
bution of effort and time required to complete course objectives. 

Attitudes toward and r eactio ns to i nstructional technology will_ 

be solicited •to substantiate and extend information from the cogni- 
tive domain. The attitudinal data relevant to the assessment 
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of educational Impact are subsumed under the major categories of 
courseware, role, and appraisal of computer-based education. Course - 
ware refers to the Instructional material employed li the demonstra- 
tions; this category encompasses content from the subject area, 
Instructional strategy, mode of delivery to the student, and procedures 
for producing Instructional materials. Role refers to attitudes and 
receptivity toward the use of computer-based Instruction. Appraisal 
concerns the priorities and basis for evaluation held by different 
audiences. These three classifications of Inquiry are Intended as 
guidelines In the development of Items for the Instruments. 

The delineation of the areas of Inquiry from broad concepts to 
specific Indicators Is Illustrated In Figure 2 by a hierarchical repre- 
sentation. As may be seen, ambiguous concepts are brought through 
successive refinements to the level of specific Issues or Item-stems. 
Priorities In the evaluation are Implied by lef t-to-r Ight positioning 
along each level of the hierarchy. Since courseware Influences 
cognitive effects and Is amenable to review by sub j ect-mat t er experts 
and Instructional psychologists, a relationship between the cognitive 
domain and courseware is depicted. Brackets enclose production to 
emphasize that responses In this area pertain only to logistical, not 
Instructional, issues. The hierarchical schema presents a partial 
elaboration of the "what" dimension in the educational analysis. 
Though Incomplete, the portrayed hierarchy is Intended to convey both 
the depth and breadth of inquiry; it also begins to specify priorities 
for the allocation of resources. Subsequent analyses will serve to 
highlight certain information, from the broad coverage, which reveals 
the strengths and weaknesses of. the respective demonstrations. 

O 
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B, Audiences, Within the context of planning the educational 
analysis, audiences (see Figure 3) are meant to denote those persons 
from whom responses will be gathered in the various areas of inquiry. 
To allow the expression of significant viewpoints potential respondents 
include representatives of the educational institution, government, 
public, and industry. Potential data sources in these sectors are: 
1) students, instructors, and administrators in the participating 
schools and colleges; 2) state and local. boards of education respon- 
sible for policies in those schools and colleges; 3) parents of 
children in the elementary schools (PLATO project only) and visitors 
to the demonstration sites; 4) committees that advise community 
colleges on the content areas; and 5) manufacturers of system compon- 
ents for PLATO and TICCIT. The focus of attention during the present 
baseline period is the educational institution, i.e. participants in 
the cooperating schools and colleges. 

Certain characteristics of students,, instructors, and administra- 
tors are relevant to identifying appropriate audiences and to esta- 
blishing proper comparison groups . The first such characteristic 
is participant status in the demonstrations. For students, partici- 
pation is determined by enrollment in classes or courses scheduled' 
to use PLATO or TICCIT instruction; for instructors, by responsibility 
for, or 8 up er vis ion of ^. CAI in course or class; for administrators, 
by responsibility f'or the implementation of the PLATO or TICCIT program. 

For those students and instructors who have no direct contacts with. 

* 

CAI, another attribute is the similarity between computer-based -and 
conventional instruction. This similarity of courses entails overlap 
in instructional objectives and resource materials. Identification 
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of students ±ix, and Instructors of, courses which parallel computer- 
bused Instruction will permit an additional comparison of Instructional 
outcomes, . ' 

Further classifications of respondents within the educational 
institution relate to the particular district, school, and course or 
department. This Information Is required to Investigate effectiveness 
across content areas, schools, and districts, and to recognize natural 
differences among audiences. Although a breakdown by districts, 
schools, and courses or departments appears applicable to students 
and Instructors, administrative positions usually satisfy only one 
of these categories according to the responsibility of an office. 

While the above delineation of the "who" dimension suffices for 
a preliminary identification, other characteristics are certainly 
important in establishing comparable or matched groups for analysis. 
Beyond descriptive information for classification, variable traits 
such as student aptitude or instructor experience might influence 
achievement or attitude. Matched groups, random assignment, and 
covariance procedures offer means to overcome anticipated control 
difficulties. The available control procedures and the multitude of 
potential respondents suggest that comparison groups for computer- 
based and conventional instruction are accessible. In addition, the 
use of baseline and concurrent comparisons for audience classifications 
provides a view of difference and change attributable to computer- 
assisted instruction* 
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C, Modes of Data Acquigitlon * The modes for acquiring data for 
the educational analysis are enumerated as: tests^ records, question- 
naires, online systems, interviews, and observations. Tests in 
standardized and objective-based form are appropriate for the instru- 
mentation of cognitive inquiries. This information on student achieve- 
ment and, possibly, aptitude will be complemented. by records and online 
systems. School records will provide additional data on achievement 
and aptitude by providing course grades and test scores; they may 
furnish supplemental information by supplying data on school attendance 
Online systems provide pertinent achievement and behavior information 
through various evaluations of student progress, and descriptions of 
student interaction and sequencing through courseware. The abundance 
of online information encompasses; performance on reviews, exercises, 
and tests; completion of required and optional materials; latencies; 
time spent at terminal. Systematic observations, especially in the 
elementary schools, mediate the accumulation of behavioral data. 

Questionnaires and interviews will constitute the primary means 
for data collection in the affective domain. Questionnaires facili- 
tate the accumulation of information on audience activities, experience 
characteristics, and opinions regarding computer-assisted instruction 
and conventional practices. Interviews provide an opportunity to 
elaborate attitudes through group and individual discussions. Possibly 
online systems can be designed to gather at t itudinal data from 
participants and site visitors. (The preceding points illustrate 
the interrelationship of instrumentation with areas of Inquiry and 
audience.) 
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D» When: Time^ of Data Acquisition, Another consideration in 
instrumentation is the time of administration. As mentioned earlier» 
baseline and concurrent control procedures are an integral part of 
the evaluation. Baseline measures permit the identification of 
reference points for achievement and attitude before the introduction 
of computer-based education. Since time differences between baseline 
and demonstration periods mighc result in a different set of respond- 
ents, concurrent comparisons between participants and non-participants 
are required to substantiate data in the educational analysis. The 
time selected for instrument administration is also crucial to the 
use of pre- and posttests as Indicators of student achievement. 

In conclusion, the evaluation for the PLATO and TICCIT projects 
will certainly address fundamental issues in computer-based education. 
Many of those issues pertain to questions which potential users might 
raise concerning practical expectations. Yet the simplest questions 
(e.g. which is better?) are perhaps the most difficult, to answer, since 
neither PLATO and TICCIT can be expected to resolve all problems in 
education or to meet the criteria of every Individual. Information, 
will also become available to answer numerous important questions 
about the impact of instructional technology within the educational 
community. The cost, technical , and educa t ional analy s es conducted 
by-EXS will identify strengths and weaknesses for PLATO and. TICCIT, 
and begin to assess, the extent to which the promise of instructional ; 
technology has been fulfilled. 



-11- 



I 






■ 










S9Jin406jnuGj{ • g 


iCa^snpui 












spjtBog itiosfApv 












SJO^TSTA V 














S4uajB£ '9 












spjBog a^B^s 


























sjorjEj^sTUfmpv •£ 


uoT^n^T^sui 
XBUoj^Bonpa 












sjo:jonjc^sm 












s:iuapn^s •! 


1. Achievement 


2. Behavior 


3. Courseware 


4. Role of CAI 


5. Appraisal 




s 


Q) 

f> 

•H 
4J 
•H 


Q) 

> 

a 

^ Q) 






- -& 

O 
C5 







(0 
Q) 

a 
a 
cu 
1-1 

4 



a 



I 



CO 
•H 

CO 

CO CO 

a CO 

0 

O 4-» 

•H U 

J' U M-i 



CO CO 

c p 

O VI 

•H P4 

0) 
0 



^4 



< M 



Q) (0 
> Q) 

H M bO 

d <u •M 

O CO 

O Ql 

M ^ 




















^ d 
•9 R 






w q 
















CD 


•rl 








d 


> 


1 4J 








rt 

M 




f d 


O 


O 


/ ^ 


& 




/ fr^ 






I 4J d 


o 




/ Q) O 




•§ 








/ d o 






/ CO 4J 



4J 3 

CO (0 



(0 

(0 d 

01 o 



(0 



i 

i 



I 



o 

(0 



d 
01 

d 



01 



ERLC 



-13- 



1 



Audiences 



BEST COPY RVMUBLE 



Educational 
Institution 




Government 



Public 





Industry 



State Local 
Boards of Education 



Parents 



Site 
Visitors 





Community 

College 
Advisory 
Committees 



Manufacturer! 
of System 
Components 



Students 
I 
I 

Participating 

status 
I 
I 
I 

Objectives and 

material 
I 
I 

District 
I 

i 

School 
I 
I 

Course or 
grade level 



Instructors 
I 
I 
I 

Participating 

status 
I 
I 

I ^ 
Objectives and District 
material 

I 
I 

District 
I 
I 
\ 

School 
I 
I 
I 

Course or 
grade level 



Administrators 
I 
I 
I 

Participating 
status 

School Division 



Department 



Figure 3 
Audience Identification 



